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ABSTRACT 

In 1992, a study was conducted at Golden West College 
(California) to determine the predictive validity of the Hath 
Diagnostic Testing Project (MDTP) tests. A total of 1,137 students 
were tested in-class; 601 took the Algebra Readiness test, 376 took 
the Elementary Algebra test, and 160 took the Intermediate Algebra 
test. Two correlation coefficients indicating predictive validity 
were computed for each course (in the first correlation withdrawals 
were converted to failing grades, and in the the second withdrawals 
were deleted from the analysis) . Only minor differences existed 
between the validity coefficients computed with the two approaches. 
The median predictive validity value across all levels was .37, which 
exceeded the standard established by the California community 
colleges. The utility of the test scores as predictors of student 
performance was also supported. A subsequent goal of the study was to 
determine how best to use the tests for making placement 
recommendations by identifying cutscores. The primary method for 
establishing cutscores involved comparing success rates for 
contrasting groups over a range of scores. The recommended cutscores 
were able to discriminate with a degree of confidence between 
students in the sample who were ultimately successful or 
unsuccessful. Data regarding factors that might also serve as 
predictors of success in mathematics courses, such as the reported 
grade earned in the student's last math course and the grade expected 
in the current course, were gathered by a questionnaire. As 
independent predictors, evidence indicated that these two indicators 
would serve approximately as well as would objective test scores. 
Attachments provide the initial student survey and a discussion of 
the adoption of revised placement rules. (JHC) 



**«****•*«*•*«»•***•«•»••••*« ****************************************** 

» Reproductions supplied by EDRS are the best that can be made * 
* from the original document. * 

****************************************************** ******* ********* * 



0 

ERIC 



CM 



Implementation and Initial Validation of the 
MDTP Tests at Golden West College 



Golden West College 
Huntington Beach, CA 



PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

S. Isonio 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER {ERIC)." 



U S Df P*fmt£N1 Of EDUCATION 

Oft** o* fcdut*t*on*t Reteirrn and impf foment 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

If Th*5 document nas oeen f«piodix;ed a*> 

r Minor change* nave been m«d« to <mpio*e 
reproduction quaMy 

• Porntb of »** o* opinions staled in in«s doc u 
roen< do f>o< nereftj^sfy 'epresent ort»c»»» 
OE Ri pos*t«on of policy 



N 
O 

N 
O 

8 



9 

ERIC 



Steven Isonio, PhD 
February, 1992 



2 

best copy miimn 



Implementation and Initial Validation of the 
MDTP Tasts at Golden Wast Col lego 

Steven Isonio, PhD 

Background/Description of the MDTP Tests 

The Math Diagnostic Testing Project (MDTP) Tests consist of 
four objectively scored, multiple choice instruments designed to 
measure mathematical skills ranging from pre-Aigebra through pre- 
Calculus. The tests offer diagnostic information in the form of 
subscores that represent particular skill areas deemed critical 
for success in targeted courses. The instruments were developed 
and are periodically revised by the Math Diagnostic Testing 
Project, a joint venture of the California State University and 
the University of California. After several years of 
development, the series of four tests was ready in 1986, and was 
then released to schools. 

The MDTP tests are currently the most widely used placement 
instruments in the California Community Colleges. Additionally, 
they are used at all of the University of California campuses, 
and approximately half of the California State Universities. 
They have received positive reviews on both practical and 
technical grounds, and were the first placement instrument to 
gain provisional approval by the Chancellor's Office for use 
beyond July 1992. 

Cux rent Math Placement Scheme at GWC and the Need for MDTP 

Assessment of mathematics skills for placement purposes at 
Golden West College is currently based upon a combination of 
information from the mathematics section of the Stanford Test of 



Academic Skills (TASK-Math) , and the Algebra test of the 
Mathematical Association of America (MAA-Algebra) . A major 
shortcoming of these instruments for placement purposes at GWC is 
that they are capable of measuring only a limited range of math 
skills — from arithmetic through elementary algebra. Because of 
this shortcoming, the highest nath class for which students can 
currently be recommended is Intermediate Algebra (Math 030) , and 
any students who "top off" the scale are not provided with a 
specific course recommendation. Therefore a distinct advantage 
of the MDTP tests is that they provide assessment of higher level 
math skills and appropriate course placement recommendations for 
these skill levels. 

An additional feature of the MDTP tests is that they offer 
diagnostic capabilities in the form of subscores with specific 
mastery levels. These subscores can be used to identify skill 
deficits in focused areas. Further, as local research on the 
tests continues, placement rules may be modified to incorporate 
these subscores into a multiple weighting scheme that leads to a 
placement recommendation. That is, for example, students who 
score below the range for recommendations into Intermediate 
Algebra on the Elementary Algebra test might be recommended to 
take Elementary Algebra if they show mastery in key. areas such as 
exponents and polynomials, whereas those students whose scores 
indicate that they have not mastered these skill areas might be 
recommended to take Pre-Algebra. Table I presents a list of the 
areas covered by each of the tests for which subscores are 
available. 



2 



Table 1 

T he MPTP Test Battery; — Number of Items and Diagnostic subscores 



Test 



Number of Iteis 



Topics 



Algebra Readines. 



50 



Integers 

Fractions 

Decimals 

Exponents/Square Roots 
Graphing 



Elementary Algebra 



50 



Arithmetic Operations 

Polynomials 

Linear Equations and 

Inequalities 
Quadratic Equations 
Graphing 

Rational Expressions 
Exponents/Square Roots 
Geometric Measurement 
Word Problems 



Intermediate Algebra 



45 



Elementary Operations 
Rational Expressions 
Exponents and Radicals 
Linear Equations and 

Inequalities 
Quadratic Polynomials 
Graphing 

Logarithms and Functions 
Word Problems 



Pre-Calculus 



40 



Rational Expressions 
Exponents and Radicals 
Linear Equations and 

Inequalities 
Polynomials and 

Polynomial Functions 
Functions 
Trigonometry 
Logarithmic and 

Exponential 

Functions 
Word Problems 



Further, the MDTP tests have received somewhat positive 
reviews by the Center for Educational Testing and Evaluation, on 
behalf of the CCC Chancellors office. The MDTP technical manual 
(MDTP, 1991) presents results from a number of predictive 
validity studies conducted at csu, UC, and Community College 
campuses. Predictive validity coefficients from these studies 
for the Algebra Readiness test range from the mid .30s to the 
high .40s. similar values are reported for the Elementary 
Algebra and Intermediate Algebra tests. Validity coefficients 
range from .33 to .61 for the Pre-Calculus test. 

Overview of the Pilot Study 

Interest in the MDTP tests has existed for years at GWC. In 
March of 1991, the GWC Math Department expressed its intention to 
eventually adopt the tests and the belief that implementation 
details and the initial system of cutscores should be 
empirically-based. Therefore the pilot study was planned during 
late Spring 1991; data collection (administration of tests and a 
brief questionnaire) took place during the Summer and Fall 1991 
sessions. Data analysis began in early 1992 when course grades 
for Fall 1991 classes were first available. 

Method 

A schedule for in-class pilot tests was developed from an 
examination of the course schedule. All testing was limited to 
the first week of classes, in order to avoid or minimize the 
effects of instruction in the class and to approximate as closely 
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as possible the sequence and timing of the assessment/placement 
process. Instructors of the selected classes were contacted and 
asked for permission to use an hour of class time to administer 
the test. Data were collected in 19 of the 20 scheduled class 
sections over the Summer and Fall sessions. A total of 1137 
students were tested during the two sessions — 601 took the 
Algebra Readiness test, 376 took the Elementary Algebra test, and 
160 took the Intermediate Algebra test. Table 2 shows the number 
of students tested, by course and by test level. 

As would be expected, there is a close correspondence 
between the number of students enrolled in the various courses 
and their representation in the pilot study data set. Since no 
data were collected for courses that would be appropriate for the 
Pre-Calculus (Level IV) test, no recommendations for its use are 
made, and implementation of this test will be delayed. 



Table 2 

Kmnfoer of Students Tested, Bv Cours e and Test Level 



Tost 

Course 


is Algebra 
Readiness 


Hi Element. 
Algebra 


Hit Inter. 
Algebra 


Elementary Algebra 
(Math 010) 


601 


— 


— 


Plane Geometry 
(Hath 020) 


mm ma 


47 


— 


Intermediate Algebra 
(Math 30) 





329 


— 


College Algebra 
(Math 115) 





— 


30 


(Math 120) 






35 


Survey of Calculus 
(Math 130) 






73 


Finite Math 
(Math 150) 






22 


TOTALS 


601 


376 


160 
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Results 

Descriptive statistics. The data collected for this pilot 
study are summarized in Table 3. These data constitute a first 
step toward the development of local norms for the MDTP t >st 
battery at Golden West College. The mean score and the 
coefficient of skewness provide a clue about the relative degree 
of difficulty of the tests, positively skewed score 
distributions indicate that sampled students tend to find the 
test difficult. Thus, the Intermediate Algebra test was found 
clearly more difficult for students in courses that it targets 
(College Algebra, Survey of Calculus, Finite Math, and 
Trigonometry) than was the Elementary Algebra test for students 
in Geometry and Intermediate Algebra. In such relative terms the 
least difficult test was Algebra Readiness, for students in 
Elementary Algebra. 

Predictive validity. The primary index of the strength of 
the relationship between test scores (predictor) and course 
grades (criterion) is the predictive validity coefficient (the 
correlation between predictor and criterion) . Two correlation 
coefficients indicating predictive validity were computed for 
each course — one between the test scores and course grades with 
all withdrawals (Ws) converted to Fail (F) , and the second 
between the test scores and course grades with withdrawals (Ws) 
deleted from the analysis. 

Table 4 presents the results of these analyses. As can be 

m 

seen, there are only minor differences between the validity 
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Table 3 

Descriptive Statistics for Tea* Score. Bv Level 



Standard 



Course 


Test 


N 


Mean 


Deviation 


Skewness 


Math 


010 


I 


600 


30.07 


7.55 


-.20 


Math 


020 


II 


47 


28.81 


9.15 


.08 


Math 


030 


II 


323 


25.98 


8.33 


.12 


Math 


115 


III 


30 


18.83 


5.52 


.26 


Math 


120 


III 


35 


23.60 


6.66 


.58 


Math 


130 


III 


72 


20.17 


7.60 


.44 


Math 


150 


III 


21 


21.24 


6.91 


.24 
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Table 4 
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Course 


Test Level 


r 


(w/)* 


r 


(w/o)* 


Math 010 


I 


.36 


(485) 


.34 


(408) 


Math 020 


II 


.45 


(35) 


.72 


(26) 


Math 030 


II 


.38 


(239) 


.39 


(202) 


Math 115 


III 


.19 


(25) 


.26 


(17) 


Math 120 


III 


.63 


(27) 


.62 


(11) 


Math 130 


III 


.36 


(54) 


.31 


(40) 


Math 150 


III 


-.09 


(20) 


.22 


(17) 



Uste: r (w/) ■ Correlation between score and grade, with M W" 
grades converted to »F" ; £ (w/o) = Correlation between score and 
grade, with "W» excluded. 
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coefficients computed for each course with the two approaches. 
With the exception of Math 150, that included only 17 students in 
the bivariate distribution that resulted in the lowest 
correlation, the median predictive validity value across all 
levels is .37, ranging from a low of .19 (for Math 115 with 
withdrawals included) to a high of .72 (for Math 020 with 
withdrawals excluded) . Minimal standards are for predictive 
validity coefficients for placement instruments to be at least 
.35 (California Community Colleges, 1991). Evidence from the 
current analysis indicates that the MDTP tests meet and even 
exceed this standard at GWC. This is a critical step in the 
local validation of assessment instruments in that without at 
least a moderate overall relationship between test scores and 
course grades, a test would be no better at generating placement 
recommendations than would a random method of doing so. Indeed, 
given the attenuation of the validity coefficient due to less 
than perfect reliability of both the predictor and criterion, and 
the restriction of range due to the use of intact groups, the 
obtained predictive validity coefficients are particularly 
impressive. 

Development of Placement Rules. Given the adequate overall 
relationship between test scores and course grades, the utility 
of the test scores as predictors of student performance is 
supported. The next issue is a very practical one— how best to 
use the tests for making placement recommendations. This 
involves identifying cutscores. Oi course, no method of 
establishing cut scores is flawless, with typical predictive 
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validities of about .35, only about 10% of the variance in 
student performance is associated with the test scores. A great 
many factors as£ tapped by the tests, both systematic and not, 
contribute to course grades. The goal of the process, therefore, 
is to improve prediction of success relative to a system that 
does not include objective test scores. 

The primary method used for establishing cutscores involved 
comparing success rates for contrasting groups over a range of 
scores. More specifically, the contrasting group-* were defined 
in what has become the standard method— the "successful »» group 
consisted of students who earned grades of A, B, or C, whereas 
persons with grades of D, F or W (withdrawal) were categorized in 
the "unsuccessful" group. Given this distinction, it was then 
possible to determine the proportion of students with specific 
scores, and hence for score ranges, that were successful. 
Candidate cutscores were then evaluated in terms of their ability 
to "predict" success (i.e., maximize the difference between 
success rates for persons scoring above that value and those 
scoring below it) . 

Table 5 details this "contrasting groups" approach for the 
case of the Algebra Readiness test (MDTP-I) for separating 
students who should be recommended to take PreAlgebra from those 
who appear ready for Elementary Algebra {that is, Math 005 versus 
Math 010) . A segment of the score range near the chosen cutscore 
is depicted in the Table. The cutscore of 28 as the minimum for 
recommendation into Elementary Algebra is associated with 47.8% 
of the students below this point being successful and 76.4% of 
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Table 5 

Analysis of Candidate Cutscores for R ecommendation of Placement 
IlXtP Elementary Algebra fMDTP TEST I — Algebra Readiness) 



Elementary Algebra Recommended for Scores >= 27 

<»26 >=27 
Successful 86 (47.5%) 273 (74.8%) 

Unsuccessful 95 (52.5%) 92 (25.2%) 

(Difference between proportions - 27.3) 



Elementary Algebra Recommended for Scores >= 28 

<=27 >=28 

Successful 97 (47.8%) 262 (76,4%) 

Unsuccessful 106 (52.2%) 81 (23.6%) 

(Difference between proportions » 28.6) 
**This cut maximizes discrimination between 
successful and unsuccessful students 



Elementary Algebra Recommended for Scores >= 2 9 

<«2C >=29 
Successful H6 (51.8%) 243 (75.5%) 

Unsuccessful 108 (48.2%) 79 (24.5%) 

(Difference between proportions - 23.7) 
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those at or above this value being successful. The difference in 
"success rates" for students above and below the cutscore is 
28.6%. Adjacent candidate cutscores produce similar values, but 
this difference is maximized by the chosen score of 28. This is 
the recommended cutscore for this case. 

Table 6 provides supplemental information in support of the 
relationship between test scores and grades. Specifically, the 
mean test scores for key criterion (grade) defined groups — for 
students earning an A or B grade, a C grade, a grade of D or F, 
and those receiving w — are presented. The mean test scores for 
the highly successful students (those with A or B grades) is 
clearly greater than the mean values for students who earned C 
grades, who in turn tended to have higher scores than those with 
D or F grades. While the pattern is clearly not a perfect linear 
relationship, these data are consistent with the overall linear 
relationships between test scores and course grades, and also 
support the reasonableness of the recommended cutscore levels. 

While scores at or above the cutscore level indicate skills 
necessary for success in the higher level course, it is not the 
case that all scores below the cutscore represent skills 
associated with success in the next lower level course. For 
example, a student who earns a very low score on the Intermediate 
Algebra test, while not recommended for Trigonometry, should 
probably .also , not be recommended for the Intermediate Algebra 
course. That is, there is a need for a "floor", of sorts, to the 
placement rule, a value below which students are either referred 
to low level courses or are referred for further evaluation. 

ERIC 13 * 5 



Table 6 

Mean Score For Key Grade-Ba sed Criterion Groups. Bv Course 



Test/ 
Course 






Grade (s) 


Earned 






A/B 


C 


D/F 


W 


Test I 












Math 


010 


33.0 


28.9 


27.7 


26.6 


Test II 












Hath 


020 


32.2 


26.2 


22.3 


27.7 


Math 


030 


28.9 


24.0 


24.2 


23.7 


Test in 












Math 


115 


20.3 


18.8 


17.0 


18.7 


Math 


120 


27.3 


* 


20.7 


21.1 


Math 


130 


24.9 


17.2 


20.9 


17.4 


Math 


150 


22.1 


* 


* 


26.3 


* Sample 


size 


for this group 


is <= 5. 
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Initial values for this "lower cutscore" were identified as the 
score that corresponds to two standard deviation units below the 
mean of the distribution of scores earned by students in the 
target course on that test. That is, if a student's score is 
lower than two standard deviation units below the mean, s/he may 
have taken the inappropriate test and should therefore be 
referred for further evaluation. 

The results of the analyses just described are presented ,n 
Table 7 in the form of recommenced placement rules. As can be 
seen, scores of 28 or above on the Algebra Readiness (I) are 
associated with the recommendation to Elementary Algebra (Math 
010) ; students with scores of 27 or lower will be directed to 
Pre-Algebra (Math 005) . For the Elementary Algebra test (II) , 24 
or greater will be associated with placement into Plane Geometry 
(Math 020) and Intermediate Algebra (Math 030) ; students with 
scores of 9 through 23 will be placed into Elementary Algebra 
(Math 010). Finally, scores of 8 or lower on this test will 
yield a recommendation to Pre-Algebra (Math 005) . The 
Intermediate Algebra test (III) will place students scoring 28 or 
higher into Trigonometry (Math 120) , 21 and 19 are the minimum 
score for placement into Survey of Calculus (Math 130) and 
College Algebra (Math 115) , respectively. The lower ranges of 
scores and associated recommendations are, 7-18 for Intermediate 
Algebra (Math 030) , and 6 or lower indicating the need for 
further evaluation (e.g., a more appropriate placement test), of 
course, students also may take courses at a level equal to ro 
lower than the highest course into which they placed. 
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Table 7 

REgQBffllEMtefl Initial Placement Rules for MDTP T ests I. n. »nH j jj 



Test 



Score Range and Recommended Placement 



Xi Algebra Readiness 

Score Range 



Recommen ded Placement 



28 - 50 
0 - 27 



Elementary Algebra (Math 010) 
PreAlgebra (Math 005) 



IX: 



Elementary Algebra 

Score Range 



24 - 50 

9-23 
0-8 



■Recommended Placement 



Plane Geometry (Math 0?0) 
Intermediate Algebra (Math 030) 

Elementary Algebra (Math 010) 

PreAlgebra (Math 005) 



XIX i Intermediate Algebra 

Scpre Range . — Recommend ed Placemen t 



28 - 45 

21 - 45 

19 - 45 
7-18 
0-6 



Trigonometry (Math 120) 
(also, Math 130, Math 115) 

Survey of Calculus (Math 130) 
(also, Math 115) 

College Algebra (Math 115) 

Intermediate Algebra (Math 030) 

Further Evaluation 
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The ability of recommended cutscores to discriminate between 
students in the sample who were ultimately successful or 
unsuccessful is summarized in Table 8a. The recommended cutscore 
for the Algebra Readiness test corresponds to the point below 
which slightly over half (52.2%) of the persons in Elementary 
Algebra (Math 010) were unsuccessful and over three- fourths 
(76.4%) of those above the cutscore are successful . The 
recommended cutscore for the Elementary Algebra test divides the 
sample of Plane Geometry (Math 020) students into those below the 
cut who have a success rate of only 28.6% and those above with a 
success rate of 76.0%. This cutscore divides the sample of 
Intermediate Algebra (Math 030) students into those below the 
value who had a success rate of 49.3% and those above with a 
success rate of 70.6%. The recommended cutscores for the courses 
associated with placements from the Intermediate Algebra test 
produce differentiations as clear as those outlined above for the 
two lower level tests. This information is detailed in Table 8b. 

Relationships Qf other Indicato r s With Gradaa . Da ta 

regarding factors that might also serve as predictors of success 
in mathematics courses were gathered by a questionnaire (see 
Appendix) . two such indicators warranting particular attention 
are the reported grade earned in the student's last math grade 
and the grade expected in the current course. Bivariate 
correlations between each of the.e variables and course grade are 
presented in Table 9 for the various levels of math courses 
sampled. There is a clear tendency for these other indicators to 
be more strongly related to course grade in the higher level 

* 17 19 



Table 8a 

Ability Of Optimal Cutscores to Discriminat e Successful/ 
Unsuccessful 



Test u par formanoe in Math oxo 

Cutscore 
<=27 >»28 
Successful 47.7% 76.4% 

Unsuccessful 52.2% 23.6% 

Test II: Performance in Math 020 

Cutscore 
<=23 >=24 
Successful 28.6% 76.0% 
Unsuccessful 71.4% 24.0% 

Test II j Performance in Math 030 

Cutscore 
<=23 >=24 
Successful 49.3% 70.6% 
Unsuccessful 50.7% 29.4% 
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Table 8b (continued) 

Ability Of QptimU Cutscores to Discriminate Successful/ 
Unsuccessful 



Test 


lilt Performance in Math 115 






Cutscore 






«=18 


>«19 




successful 53.8% 


78 . 6% 




Unsuccessful 46.2% 


21.4% 


Test 


Ills Performance in Math 120 






cutscore 






<«27 


>=28 




Successful 17.4% 


87.5% 




Unsuccessful 82.6% 


12.5% 


Test 


lilt Performance in Math 130 






Cutscore 






<=20 


>=21 




Successful 45.2% 


73.3% 




Unsuccessful 54.8% 


26.7% 
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Table 9 

RelaUPDSlUPg Between Reported Last Math Grade and Current Grade. 
and Expected Grade and Curr ent Grade 



Course Test r. (LstGrade) £ (ExpGrade) 



Math 


010 


I 


.08 


.13 


Math 


020 


II 


.51 


.22 


Math 


030 


II 


.42 


.47 


Math 


115 


III 


.57 


.23 


Math 


120 


III 


.30 


.29 


Math 


130 


III 


.43 


.46 


Math 


150 


III 


.31 


. 33 



Hafcg: £ (LstGrade) refers to correlation between the reported 
grade earned in the student's last math class and the grade 
earned in the current class. r. (ExpGrade) refers to correlation 
between the grade the expected to earn in current class, and the 
actual grade earned. 



20 

22 



classes. Further, the overall average relationship with course 
grade is slightly stronger for the last math grade than for the 
expected grade, but the difference is not sizeable (median 
correlations across courses are .42 for the former and .33 for 
the latter). It is important to note that, as independent 
predictors, evidence indicates that these two indicators would 
serve approximately as well as would objective test scores. 

Prediction based upon multiple indicators, each at least 
moderately correlated with the criterion, is typically superior 
to prediction based upon a single indicator. The positive 
findings concerning the relationships between both last math 
grade and expected grade and course grade suggest that these and 
similar variables may be useful as additional predictors in 
future, more comprehensive, placement models. 

As of December 1991, with the implementation of the 
Computerized Assessment and Placement Program (CAPP) , the GWC 
Assessment Center now can regularly gather, store locally, and 
easily retrieve a broad range of information in addition to test 
scores. Candidate predictors such as previous experiences math 
courses, expectations, and other variables can be evaluated 
concerning their appropriateness as predictors of success in math 
classes. Regression-based, multiple indicator placement rules 
can be developed for the full range of mathematics courses. As 
noted earlier, some of the diagnostic subscores might also be a 
part of these models, m any case, some of the data collected 
with the questionnaire for this pilot study suggest that, as more 
data are collected and the necessary analyses are performed, 
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effective multiple indicator models can certainly be developed. 

summary and Discussion 

In summary, there is clear evidence of the predictive 
validity of the MDTP tests at GWC. The median value for these 
coefficients for courses that will be a part of initial placement 
rules is approximately .37. This value exceeds the minimal 
standard for predictive validity of assessment tests for 
placement purposes in California Community Colleges, sets of 
placement rules for test levels I, II, and III were developed. 
A variety of lines of evidence supporting the reasonableness of 
these placement recommendations were presented in this report. 

Further, it is critical to point out that the placement 
rules developed and outlined above are recommendations for 
initial implementation of the MDTP tests at Golden West college. 
They will be monitored and adjusted as needed. Additionally, a 
number of followup and supplemental studies are required. There 
is a need to evaluate the extent to which disproportionate impact 
results from the use of these tests. A related issue that must 
be investigated is that of differential validity. Specifically, 
this issue involves the relative validity of the tests as 
placement tools for a number of important subgroups. Finally, as 
outlined above, models with multiple predictors will be developed 
and the optimal use of diagnostic subscore information must be 
determined. 
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MATH DIAGNOSTIC TESTING PROJECTS PILOT STUDY 

Student Survey 



1. How many years of math did you complete in high school? 

a. less than one d. three 

b. one e. four 

c. two 



2. How many math courses have you completed in college? 

a. none d. three 

b. one e. four or more 

c. two 



3. What was the last math class that you completed? (Please 
indicate the name, not the number of the class). 



4. What grade did you receive in the last math class that you 
completed? 

a. A d. D 

b. B e. Withdrawal or incomplete 

c. c f. F 



5. What grade do you expect to earn in this math class? 



Your Social Security Number: 




Implementation and initial Validation of the 
MDTP Tests at Golden west Colleges 
Supplement 

Steven Isonio, PhD 

Background . 

The process of evaluating predictive validity and developing 
placement rules for the Math Diagnostic and Testing Project 
(MDTP) instruments was described in the primary report. The 
study on which that report was based yielded recommendations for 
initial placement rules for three of the tests in the battery. 
Data were gathered and analyzed that are relevant to these 
placement recommendations involving three tests of the four MDTP 
tests, and seven mathematics courses at GWC. Implementation of 
the fourth MDTP test, and assessment for placement into higher 
level math courses will be will be delayed, pending the 
collection and analysis of additional data. 
Subsequent DeveXop?n?nrff 

During two meetings to discuss MDTP implementation with the 
GWC Math Department (in early February 1992), some concerns were 
expressed that the recommended cut scores for differentiating 
Math 005 from Math 010 placements (with the Algebra Readiness 
test) and between Math 010 and Math 030/Math 020 placements (with 
the Elementary Algebra test) might be too high. These concerns 
were largely based upon observations that while the proportion of 
successful students above the cutscore was in the mid-70% ranee, 
it was nearly 50% for the group below, the cutscore. The 
expectation was that the placement rule should result in a 
clearer majority of students below the cucscore who are 



unsuccessful. Subsequent analyses revealed that the ability to 
discriminate between successful and unsuccessful students was not 
markedly poorer for scores just below the values originally 
recommended for Math 010 and Math 030 (see Table 1). That is, 
in the case of Math 010, a cut score of 25/26 (instead of 27/28) 
separates the sample into an upper group that has a success rate 
of 73.2% and a lower group with a success rate of only 46.0%. in 
the case of Math 030, the success rates for these two groups 
(defined by a 21/22 cut instead of 23/24) is 67.5% for the upper 
group and 50.9% for the lower group. 

Since the overall pass rates were in the mid-60%'s for the 
classes in the sample, contingency tables were developed that 
excluded students earning «c» grades from the "success" group to 
assess the ability of the cutscores to differentiate between 
these two contrasting groups (see Table 2). This analysis 
revealed that the new cutscores are clearly capable of making the 
differentiation, 

In light of these analyses, and since lower cutscores make 
it possible to monitor the performance of students in the band of 
scores recommended to the higher level course, the GWC Math 
department decided to adopt the placement rules depicted in 
Table 3. Students scoring in the lower region of the upper 
placement can be closely monitored; if their performance in class 
is markedly lower than that of students in the higher regions, 
the cutscore can be lowered. Conversely, if their success rate 
is comparable to the others in the range, the cut score can be 
maintained. 
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Summary and Recntn^ondatlon*. 

The primary report and this supplement document the process 
of pilot testing the MDTP tests at Golden West College, strong 
evidence for predictive validity is presented and the development 
of placement rules based upon a combination of empirical and 
judgmental approaches is described. 

Additional research is needed in the following areas i 

—Cross validation with separate samples. 

-Analysis of differential validity and disproportionate 
impact in placement recommendations. 

—Development of a comprehensive placement model based upon 
multiple measures 

—Monitor rates at which students select the various MDTP 
tests as well as placement rates to assist planning and 
scheduling 

-Develop a placement rule and establish predictive validity 
for MDTP test number 4, Pre-Calculus 
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Table l 

Analyses qf New Cut Scores For M»th 010 and 0 ™ 



Math 010 — Algebra Readiness Test 



Outcome 




25 


>=26 


Successful 


46.0% 


73.2% 


Not 
Successful 


54.0% 


26.8% 



Math 030 — Elementary Algebra Test 



Outcome 


Cutscores 


<= 21 


>«22 


Successful 


50.9% 


67.5% 


Not 
Successful 


49. 1% 


32.5% 
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Table 2 



With «c» qr*d ? s flxclnrtPd From Sn ^ ssful r , rn „ p 



Math 010 — Algebra Readiness Test 



Outcome 



Cutscores 



<« 25 



>«26 



Successful 



26.4% 



65.7% 



Mot 
Successful 



73.6% 



34.3% 



Math 030 — Elementary Algebra Test 



Outcome 



Successful 

Not 
Successful 



Cutscores 



<« 21 



36.9% 



63.1% 



>*»22 



66.7% 



33.3% 
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Table 3 

Placement Rules for MDTP Tests J. n. and Trr r 
As Adopted 



Test 



Score Range and Recommended Placement 



It 



lis 



Algebra Readiness 

Score Range 



Recommen ded Placemen t 



26 - 50 
0-25 



Elementary Algebra (Math 010) 
PreAlgebra (Math 005) 



Elementary Algebra 

Score Ra^ce Recommended Placement, 



22 - 50 

9-21 
0-8 



Plane Geometry (Math 020) 
Intermediate Algebra (Math 030) 

Elementary Algebra (Math 010) 

PreAlgebra (Math 005) 



III: intermediate Algebra 

Score Rapqe Recommen ded Placem ent 



28 - 45 

21 - 45 

19 - 45 
7-18 

0-6 



Trigonometry (Math 120) 
(also, Math 130, Math 115) 

Survey of Calculus (Math 130) 
(also, Math 115) 

College Algebra (Math 115) 

Intermediate Algebra (Math 030) 
Plane Geometry (Math 020) 

Further Evaluation 
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